Information processing apparatus, information processing method, and program

ABSTRACT

There is provided an information processing apparatus including an emission unit for emitting a projection image onto a subject, which is to be photographed, so as to allow the subject to perform a predetermined operation, a recognizing unit for recognizing operation performed by the subject, and an execution unit for executing a predetermined processing according to the operation of the subject that is recognized by the recognizing unit.

CROSS-REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/340,382 (filed on Nov. 1, 2016), which is a continuation of U.S.patent application Ser. No. 14/711,092 (filed on May 13, 2015 and issuedas U.S. Pat. No. 9,516,206 on Dec. 6, 2016), which is a continuation ofU.S. patent application Ser. No. 13/023,639 (filed on Feb. 9, 2011 andissued as U.S. Pat. No. 9,049,376 on Jun. 2, 2015), which claimspriority to Japanese Patent Application No. 2010-048098 (filed on Mar.4, 2010), which are all hereby incorporated by reference in theirentirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an information processing apparatus, aninformation processing method, and a program.

Description of the Related Art

In the past, a flash is generally used as illumination when an image iscaptured with a camera. However, it used to be difficult to performspecial illumination other than the flash with only the flash. Withimage processing, a desired pattern, a texture, and the like can beapplied to an image captured by a camera. However, although aphotographer who uses the camera can see the result of image processing,a third person and a subject may not see the result of image processing.

For example, Japanese Translation of PCT Application No. 2006-523067 andJapanese Patent Application Laid-Open No. 2009-76983 disclose techniquesfor causing a projector to project a graphic image onto a subject who isphotographed. For example, Japanese Patent Application Laid-Open Nos.2008-152622 and 2009-70245 disclose a technique for projecting a graphicimage onto a desk and the like, detecting a direction of fingers andshapes of hands, and causing an information processing apparatus toperform predetermined processing.

SUMMARY OF THE INVENTION

However, it is desired to not only perform processing according to adirection of fingers and shapes of hands but also detect a certainmovement of a subject with respect to a projected graphic image andexecute processing in association with the detected movement.

In view of the foregoing, it is desirable to provide a novel andimproved information processing apparatus, information processingmethod, and program, wherein processing can be executed according tosubject's interaction with a projected image.

According to an embodiment of the present invention, here is provided aninformation processing apparatus including an emission unit for emittinga projection image onto a subject, which is to be photographed, so as toallow the subject to perform a predetermined operation, a recognizingunit for recognizing operation performed by the subject, and anexecution unit for executing a predetermined processing according to theoperation of the subject that is recognized by the recognizing unit.

The recognizing unit may recognize the operation of the subject based onan image of the subject captured by an image capturing device.

The emission device may emit the projection image onto the subject, andthe image capturing device and the emission device may be arranged atfixed positions of the information processing apparatus.

The emission unit may emit the projection image for allowing the subjectto perform selection operation, the recognizing unit may recognize theselection operation performed by the subject, and the execution unit mayexecute operation selected by the subject.

The emission unit may emit the projection image selected by user'soperation onto the subject.

The recognizing unit may recognize an object and a person included inthe subject, and the emission unit may emit the projection image onto aregion of the object and the person included in the subject, which isrecognized by the recognizing unit.

The recognizing unit may recognize movement of the subject in anemission range onto which the projection image is emitted.

The execution unit may execute a predetermined processing correspondingto movement of the subject in the emission range recognized by therecognizing unit.

The execution unit may cause the emission unit to emit a projectionimage according to operation of the subject recognized by therecognizing unit.

According to another embodiment of the present invention, here isprovided an information processing method including the steps ofemitting a projection image onto a subject, which is to be photographed,so as to allow the subject to perform a predetermined operation,recognizing operation performed by the subject, and executing apredetermined processing according to the operation of the subject thatis recognized by the recognizing unit.

According to another embodiment of the present invention, here isprovided a program for causing a computer to function as an informationprocessing apparatus including an emission unit for emitting aprojection image onto a subject, which is to be photographed, so as toallow the subject to perform a predetermined operation, a recognizingunit for recognizing operation performed by the subject, and anexecution unit for executing a predetermined processing according to theoperation of the subject that is recognized by the recognizing unit.

As described above, according to the present invention, processing canbe executed according to subject's interaction with a projected image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating an overview of anembodiment of the present invention;

FIG. 2 is an explanatory diagram illustrating an overview of anembodiment of the present invention;

FIG. 3 is a block diagram illustrating a hardware configuration of aninformation processing apparatus according to a first embodiment of thepresent invention.

FIG. 4 is a block diagram illustrating a functional configuration of acontrol unit according to the embodiment;

FIG. 5 is an explanatory diagram illustrating an example of a projectionimage emitted according to the embodiment;

FIG. 6 is an explanatory diagram illustrating an example of a projectionimage emitted according to the embodiment;

FIG. 7 is an explanatory diagram illustrating an example of a projectionimage emitted according to the embodiment;

FIG. 8 is an explanatory diagram illustrating an example of a projectionimage emitted according to the embodiment;

FIG. 9 is an explanatory diagram illustrating an example of a projectionimage emitted according t the embodiment;

FIG. 10 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 11 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 12 is an explanatory diagram illustrating an example of aprojection image emitted according t the embodiment;

FIG. 13 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 14 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 15 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 16 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 17 is a flowchart illustrating operation of the informationprocessing apparatus according to the embodiment in detail;

FIG. 18 is a flowchart illustrating operation of the informationprocessing apparatus according to the embodiment is detail;

FIG. 19 is a block diagram illustrating a hardware configuration of aninformation processing apparatus according to a second embodiment of thepresent invention;

FIG. 20 is a block diagram illustrating a functional a configuration ofa control unit according to the embodiment;

FIG. 21 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 22 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 23 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 24 is an explanatory diagram illustrating a example of a projectionimage emitted according to the embodiment;

FIG. 25 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 26 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 27 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 28 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 29 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 30 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 31 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 32 is a flowchart illustrating operation of the informationprocessing apparatus according to the embodiment in detail;

FIG. 33 is a flowchart illustrating operation of the informationprocessing apparatus according to the embodiment in detail;

FIG. 34 is a flowchart illustrating operation of the informationprocessing apparatus according to the embodiment in detail;

FIG. 35 is an explanatory diagram illustrating an overview of a thirdembodiment of the present invention;

FIG. 36 is an explanatory diagram illustrating it overview of theembodiment;

FIG. 37 is a block diagram illustrating a functional configuration of acontrol unit according to the embodiment;

FIG. 38 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 39 is an explanatory diagram illustrating an example of aprojection image entitled according to the embodiment;

FIG. 40 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 41 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 42 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 43 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 44 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 45 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 46 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 47 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 48 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment;

FIG. 49 is an explanatory diagram illustrating an example of aprojection image emitted according to the embodiment; and

FIG. 50 is a flowchart illustrating operation of an informationprocessing apparatus according to the embodiment in detail.

DETAILED DESCRIPTION OF THE EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

-   <1. Overview of the present embodiment>-   <2. First embodiment>-   <2-1. Hardware configuration of information processing apparatus>-   <2-2. Functional configuration of information processing apparatus>-   <2-3. Operation of information processing apparatus in detail>-   <3. Second embodiment>-   <3-1. Hardware configuration of information processing apparatus>-   <3-2. Functional configuration of information processing apparatus>-   <3-3. Operation of information processing apparatus in detail>-   <4. Third embodiment>-   <4-1. Hardware configuration of information processing apparatus>-   <4-2. Functional configuration of information processing apparatus>-   <4-3. Operation of information processing apparatus in detail>

1. Overview of the Present Embodiment

First, an overview of the present embodiment will be explained withreference to FIG. 1 and FIG. 2. The information processing apparatus 10according to the present embodiment mainly includes an image capturingdevice (camera) for capturing an image of a subject, an emission device(projector) for projecting a video image onto the subject, a displaydevice (display) for displaying the captured image taken by the camera,and the like. As shown in FIG. 1, one face (front surface) of theinformation processing apparatus 10 is arranged with a display 11. Theother face (back surface) of the information processing apparatus 10 isarranged with a camera lens 12 and a projector lens 13. In theexplanation below, the camera including the camera lens 12 is referredto as an image capturing device 12 or a camera 12. The projectorincluding the projector lens 13 is referred to as an emission device 13or a projector 13.

In the information processing apparatus 10, the camera 12 and theprojector 13 are arranged at fixed positions of the casing of theinformation processing apparatus 10. An image of a subject captured bythe camera 12 is displayed on the display 11. The projector 13 projectsa projection image onto the subject. Then, the display 11 displays thesubject onto which the projection image is projected.

As shown in FIG. 2, a photo shooting region 51 is a region that can becaptured by the camera. A projection region 52 is a region onto whichthe projector can emit light. In the explanation about the embodimentsbelow, the photo shooting region 51 and the projection region 52 areconsidered to be in synchronization. In this case, “being insynchronization” means that the photo shooting region 51 captured by thecamera lens 12, i.e., a display region displayed on the display 11, isthe same as the projection region 52 onto which light is mined by theprojector lens 13. When the photo shooting region 51 and the projectionregion 52 are not in synchronization, the display 11 may indicate theprojection region. The overview of the present embodiment has beenhereinabove explained.

2. First Embodiment

Subsequently, the first embodiment will be explained. In the past, aflash is generally used as illumination when an image is captured with acamera. However, it used to be difficult to perform special illuminationother than the flash with only the flash. With image processing, adesired pattern, a texture, and the like can be applied to an imagecaptured by a camera. However, although a photographer who uses thecamera can see the result of image processing, a third person and asubject may not see the result of image processing.

Accordingly, there are disclosed techniques for causing a projector toproject a graphic image onto a subject who is photographed. In thesetechniques, a position to which a graphic image is to be projected isrecognized using a marker and the like in advance, and the graphic imageis projected onto the marker. In the above techniques, however, thegraphic image is projected onto a position recognized using the markerand the like in advance. Therefore, the above techniques have an issuein that the graphic image may not be projected in such a manner as toenable illumination expression with a high degree of flexibility, e.g.,using a graphic image projected onto all over a photographed target asillumination and projecting a graphic only onto a specified position.

Accordingly, the information processing apparatus 10 according to theembodiment of the present invention has been created in view of theabove circumstance as one of issues taken into consideration. With theinformation processing apparatus 10 according to the present embodiment,a graphic image suitable for a photographed image can be projected.

2.1 Hardware Configuration of Information Processing Apparatus

Subsequently, a hardware configuration of the information processingapparatus 10 will be explained with reference to FIG. 3. FIG. 3 is ablock diagram illustrating the hardware configuration of the informationprocessing apparatus 10. As shown in FIG. 3, the information processingapparatus 10 includes a display device (display) 11, an image capturingdevice (camera) 12, an emission device (projector) 13, a CPU 14, aninput device 15, a RAM (Random Access Memory) 16, a nonvolatile memory17, and the like.

The CPU 14 functions as a calculation processing unit and a controldevice, so as to control overall operation in the information processingapparatus 10 according to various kinds of programs. The CPU 104 may bea microprocessor. The RAM 16 temporarily stores programs used duringexecution of the CPU 14 and parameters and the like that change asnecessary during the execution. These are connected with each other by ahost bus constituted by a CPU bus and the like. The nonvolatile memory17 stores programs, calculation parameters, and the like used by the CPU14. The nonvolatile memory 17 may be, for example, a ROM (Read OnlyMemory), a flash memory, and the like.

The display device 11 is an example of an output device for outputtinginformation. The display device 11 may be, for example, a liquid crystaldisplay (LCD) device, and an OLED (Organic Light Emitting Diode) device.

The image capturing device 12 has a function of causing a CCD to convertlight obtained through an image capturing lens into an electric signalso as to convert an analog signal into a digital signal, thus capturingan image of a subject. The image taken by the image capturing device 12is displayed on the display device 11.

The emission device 13 has a function of emitting light onto a subjectthrough an emission lens. The emission device 13 projects apredetermined projection image onto a subject according to control ofthe CPU 14. As described above, the image capturing device 12 and theemission device 13 are arranged a fixed positions in the informationprocessing apparatus 10. The following explanation will be given underan assumption that the photo shooting region captured by the imagecapturing device 12 and the projection region onto which light isemitted by the emission device 13 are in synchronization, in otherwords, they are in the same region.

The input device 15 includes, for example, input means for a user toinput information such as a touch panel, buttons, switches, and levers,and an input control circuit for generating an input signal based onuser input and outputting the input signal to the CPU 14.

2-2. Functional Configuration of Information Processing Apparatus

The hardware configuration of the information processing apparatus 10has been hereinabove explained. Subsequently, a functional configurationof a control unit (CPU 14) of the information processing apparatus 10will be explained with reference to FIG. 4. Reference is made to FIGS. 5to 16 as necessary when the functional configuration of FIG. 4 isexplained. FIGS. 5 to 16 are explanatory diagrams each illustrating anexample of a projection image emitted. FIG. 4 is a block diagramillustrating a functional configuration of the control unit of theinformation processing apparatus 10.

As shown in FIG. 4, the control unit of the information processingapparatus 10 includes an analyzing unit 102, an emission unit 104, agenerating unit 106, a storage unit 108, and the like. The analyzing nut102 has a function of analyzing an image of a subject captured by theimage capturing device 12. The analyzing unit 102 analyzes the image ofthe subject, and recognizes an image of an object included in thesubject. In addition, the analyzing unit 102 detects an image of aperson included in the subject by analyzing the image of the subject,and defects the image of a face. In addition, the analyzing unit 102analyzes the image of the subject, and detects the color of the subject.An analysis result of the image of the subject analyzed by the analyzingunit 102 is provided to the emission unit 104 and the generating unit106.

The emission unit 104 has a function of emitting a projection imageusing the emission device 13 according to the analysis result of theimage of the subject analyzed b the analyzing unit 102. The storage unit108 stores a projection image corresponding to the analysis result ofthe image of the subject. The emission unit 104 obtains the projectionimage corresponding to the analysis result of the image provided by theanalyzing unit 102 from the storage unit 108, and emits the projectionimage to the subject using the emission device 13.

For example, as shown in FIG. 5, the image capturing device 12 capturesan image of the subject 401, and recognizes an image 402 of the subject.Then, a projection image 403 is selected according to the image 402 ofthe recognized subject, and the projection image 403 is emitted onto thesubject by the emission device 13. In FIG. 5, the face included in thesubject is recognized, and the projection image for illuminating thefacial portion is selected. Then, the image capturing device 12 capturesan image of a subject 405 onto which the projection image 403 isprojected, and a display image 404 is displayed on the display device11.

Explanation will be continued with reference back to FIG. 4. Forexample, when the analyzing unit 102 recognizes an image of an objectincluded in the subject, a projection image for the image of the objectis obtained from the storage unit 108 and is emitted onto the subject.Alternatively, when an image of a person or an image of a face includedin the subject is detected, a projection image for the image of theperson or the face is obtained from the storage unit 108 and is emittedonto the subject. On the other hand, when the image of the subject isanalyzed, and a particular color included in the subject is detected, aprojection image for the particular color is obtained front the storageunit 108 and is emitted onto the subject.

The generating unit 106 has a function such that, when the storage unit108 stores a predicted image in advance for predicting an image where aprojection image is emitted, the generating unit 106 generates aprojection image according to a comparison result obtained by comparingthe predicted image with an emission image onto which the projectionimage is projected by the emission unit 104. The projection imagegenerated by the generating unit 104 is emitted by the emission unit104, and the projection image generated by the emission unit 104 isemitted onto the subject.

As described above the storage unit 108 stores the analysis result ofthe image of the subject and the projection image, which are associatedwith each other. In addition, the storage unit 108 stores a predictedimage in advance for predicting an image onto which a stored projectionimage is emitted.

An example of a projection image emitted by the emission device 13 willbe hereinafter explained with reference to FIGS. 6 to 16. FIG. 6 shows acase where a projection image is projected onto an object having aparticular color so that a color different from the particular color isapplied to the object. In FIG. 6, for example, the subject includes aplurality of white flowers. First, the analyzing unit 102 detectsobjects 406 having the particular color included in the subject 405. Asshown in an explanatory diagram 406, a plurality of “white flowers” inwhich a white region has a certain size or more is detected. Then, theprojection image according to the shapes of the objects 406 and thepositions of the objects 406 is generated, and the generated projectionimage is projected onto the subject. As shown in an explanatory diagram408, a projection image having a color different from white such as pinkis generated and is emitted onto the regions of the “white flowers”,Therefore, a color different from the original color is emitted onto thesubject, and the subject onto which the projection image is projectedcan be captured as an image.

As shown in FIG. 7, when the display screen is a touch panel, an objectincluded in the subject onto which a projection image is emitted may beselected by user's input. As shown in an explanatory diagram 412, whenthe “white flowers” are selected by user's input, objects in which awhite region has a certain size or more are selected. Objects having thesame shape or the same color as the object selected by user's input mayalso be selected at the same time. Then, the projection image accordingto the shapes of the objects 412 and the positions of the objects 412 isgenerated, and the generated projection image is projected onto thesubject. As shown in an explanatory diagram 413, a projection imagebasing a color different from white such as pink is generated and isemitted onto the region of the “white flowers” selected by a user.

FIG. 8 is an explanatory diagram illustrating an internal processingwhen the projection images shown in FIGS. 6 and 7 are projected. Asshown in FIG. 8, first, the image capturing device 12 captures an imageof a subject, and analyzes regions such as objects and colors includedin the captured image. Then, a projection image is generated used on theanalysis result. In the example shown in FIG. 8, objects in which awhite region has a certain size or more are detected from the capturedimage, and a projection image is generated such that a projection coloris arranged so as to exert effect only on the pixels constituting theobjects and black pixels are arranged for pixels constituting thoseother than the objects in question. Then, as a result of emission of thegenerated projection image, the projection color is emitted ontoportions of the subject in which a white region has a certain size ormore.

FIG. 9 shows an example where a particular region is detected, and aprojection image such as a pattern and texture is emitted onto thedetected region. As shown in FIG. 9, an image 417 captured by the imagecapturing device 12 is analyzed, and a face included a subject isdetected. Then, as shown in the explanatory diagram 417, positions of abody and clothes are detected based on an arrangement between the sizeof the face and the size of the subject. Then, as shown in anexplanatory diagram 418, a projection image to be emitted onto theregions at the positions of the body and the clothes is generated. Inthe explanatory diagram 418, a projection image for emitting a flowerpattern onto the region of the clothes is generated. Then, as shown inan explanatory diagram 420, the flower pattern is emitted onto theregion of the clothes of the subject.

Now, the internal processing for emitting the projection image shown inFIG. 9 will be explained with reference to FIG. 10. As shown in anexplanatory diagram 421 of FIG. 10, first, a face included in a subjectis recognized from an image captured by the image capturing device 12.Then, as shown in an explanatory diagram 422, a body is estimated from arecognition result of a face. Further, as shown in an explanatorydiagram 423, a possible projection region is estimated from theestimated region of the body. In this case, the white region in the bodyis adopted as the possible projection region. Then, a projection imageis generated so as to project a predetermined pattern onto the possibleprojection region estimated in the explanatory diagram 423. Then, asshown in an explanatory diagram 425, the generated projection image isemitted onto the subject.

Subsequently, an example will be explained with reference to FIG. 11, inwhich a particular object is detected, and a projection image is emittedavoiding the detected object. As shown in an explanatory diagram 426 ofFIG. 11, first, a face included in a subject is recognized from an imagecaptured by the image capturing device 12. Then, a projection imageshown in an explanatory diagram 427 is generated to project apredetermined pattern onto a region other than the recognized region ofthe face. Then, as shown in an explanatory diagram 428, the generatedprojection image is emitted onto the subject.

Subsequently, a case will be explained with reference to FIG. 12, wherea projection image is generated using a predicted image. As describedabove, when the storage unit 108 stores the predicted image, aprojection image can be generated according to a comparison resultobtained by comparing the predicted image with the emission image ontowhich the projection image is projected. As shown in an explanatorydiagram 430 of FIG. 12, first, the image capturing device 12 captures animage of a subject, and analyzes the image of the subject. Then, asshown in an explanatory diagram 431, a projection image having apredetermined odor is emitted onto the subject.

Then, the predicted image stored in the storage unit 108 and theemission image as shown in the explanatory diagram 431 are compared, anda difference therebetween is detected. A projection image as shown in anexplanatory diagram 432 is generated according to the difference. Forexample, in the explanatory diagram 431, a projection image capable ofoutputting yellow color effect is emitted onto the subject. The yellowcolor effect caused by the predicted image and the yellow color effectof the emission image are compared. When the yellow color effect exertedon the mission image is strong, a projection image exerting a lighteryellow color effect is generated. Then, as shown in an explanatorydiagram 433, the generated projection image is emitted onto the subject.

Subsequently, emission of a projection image where a subject is a facewill be explained with reference to FIG. 13. As shown in FIG. 13, first,an image captured by the image capturing device 12 (explanatory diagram435) is analyzed, and is shown in an explanatory diagram 436, thepositions of the mouth and the eyes are detected in the face. Then, aprojection image to be emitted onto the face is obtained from thestorage unit 108, and a projection image as shown in an explanatorydiagram 437 is generated. In the explanatory diagram 437, the positionsa cheeks in the face are identified from the positions of the mouth andthe eyes in the face, and a projection image is generated to emit arouge color such as pink onto the positions of the cheeks in the face.Then, as shown in an explanatory diagram 438, the generated projectionimage is projected onto the subject.

Subsequently, emission of a projection image according to a brightnessof a captured image will be explained with reference to FIG. 14. Asshown in FIG. 14, first, an image captured by the image capturing device12 (explanatory diagram 440) is analyzed, and as shown in an explanatorydiagram 441, a highly bright region is detected. Then, when a regionhaving a brightness equal to or higher than a certain level is detected,a projection image is generated according to the level of thebrightness. In an explanatory diagram 441, a region around a candle of acake is detected as a region having a brightness higher than otherregions. Then, as shown in an explanatory diagram 442, the generatedprojection image is projected onto the subject. In the explanatorydiagram 442, a star pattern is emitted to a region around the candle ofthe cake detected as the highly bright region.

Subsequently, emission of a projection image according to movement of atarget to be photographed will be explained with reference to FIG. 15.As shown in FIG. 15, movement of a subject is detected from a motionpicture captured by the image capturing device 12. For example, themovement of the subject may be detected by detecting change of pixelvalues. Then, a projection image is generated according to the movementof the detected object. As shown in an explanatory diagram 445, thesubject running to the left is detected, and a star pattern is emittedin the direction the subject is running. The projection image accordingto the movement of the object is emitted while the projection image ismoved together with the subject who is running in the direction in whichthe subject is running.

As shown in FIG. 16, the analyzing unit 102 analyzes an image capturedby the image capturing device 12. When it is determined that a subjecthas no region onto which a projection image can be projected, a messagemay be presented to tell that there is on possible projection region. Anexample of a case where a subject does not have a region onto which aprojection image can be projected is that the background color is blackand the like, and even when the projection image is emitted, theprojection image may not be formed on the subject. For example, as shownin an explanatory diagram 450, when the background of the captured imageis black, a dialogue “there is no possible projection region” isdisplayed as shown in an explanatory diagram 451.

2-3. Operation of Information Processing Apparatus at Detail

The functions of the control unit of the information processingapparatus 10 have been hereinabove explained. Subsequently, operation ofthe information processing apparatus 10 will be explained in detail withreference to FIGS. 17 to 18. FIGS. 17 and 18 are flowcharts eachillustrating operation of the information processing apparatus 10 indetail.

As shown in FIG. 17, first, the information processing apparatus 10determines whether a shooting mode is a projection shooting mode ontowhich a projection image can be emitted (S102). When the shooting modeis determined to be the projection shooting mode in step S102, an imagecaptured by the image capturing device 12 is obtained (S104).

Then, the analyzing unit 102 recognizes objects included in the imageobtained in step S104 (S106). Then, a determination is made as towhether the objects recognized in step S106 include any target objectonto which a projection image can be emitted (S108). The determinationin step S108 can be made by determining whether the storage unit 108stores a projection image corresponding to the object recognized in stepS106. Further, whether the projection image can be emitted may bedetermined by detecting a background color of the image.

When the subject is determined to include a target object onto which aprojection image can be emitted in step S108, the projection imagecorresponding to the target object is obtained from the storage unit108, and a projection graphic image is generated (S110). Then, theprojection graphic image generated in step S110 is emitted onto thesubject (S112). When the subject is determined not to include a targetobject onto which a projection image can be emitted in step S108, adialogue is displayed on the display screen to tell that there is noprojection target (S114).

Then, a determination is made as to whether photo shooting is performedor not (S116). When it is determined that photo shooting is performed instep S116, the captured image is saved (S118). When it is determinedthat photo shooting is not performed in step S116, processings of stepS102 and subsequent steps are repeated. The processings shown in theflowchart of FIG. 17 have been hereinabove explained. Subsequently,processings as shown in the flowchart of FIG. 18 will be explained.

FIG. 18 is a flowchart illustrating operation of the informationprocessing apparatus 10 in detail where the storage unit 108 stores apredicted image. As shown in FIG. 18, first, the information processingapparatus 10 determines whether a shooting mode is a projection shootingmode in which a projection image can be emitted (S122). When theshooting mode is determined to be the projection shooting mode in stepS122, an image captured by the image capturing device 12 is obtained(S124).

Then, the analyzing unit 102 recognizes objects included in the imageobtained in step S104 (S126). Then, a determination is made as towhether the objects recognized in step S126 include any target objectonto which a projection image can be emitted (S128). When the subject isdetermined to include a target object onto which a projection image canbe emitted in step S128, the projection image corresponding to thetarget object is obtained from the storage unit 108, and a projectiongraphic image is generated (S130). Then, a projection graphic imagegenerated in step S130 is emitted onto the subject (S132.). In thiscase, step S136, it is considered that the storage unit 108 stores apredicted image for predicting an image where a desired projection imageis emitted onto the object recognized in step S126.

Then, an image of a subject onto which a graphic image is projected instep S132 is obtained (S134). Then, it is determined that the emissionimage obtained in step S134 has become similar to the predicted imagestored in the storage unit 108 (S136). When it is determined that theemission image has become similar to the predicted image in step S136,the processing of step S140 is executed. When it is determined that theemission image has not become similar to the predicted image in stepS136, the processings of step S130 and subsequent steps are repeated.

When the subject is determined not to include a target object onto whicha projection image can be emitted in step S128, a dialogue is displayedon the display screen to tell that there is no projection target (S138).

Then, a determination is made as to whether photo shooting is performednot (S140). When it is determined that photo shooting is performed instep S140, the captured image is saved (S142). When it is determinedthat photo shooting is not performed in step S140, the processings ofstep S122 and subsequent steps are repeated. The processings shown inthe flowchart of FIG. 18 have been hereinabove explained.

As described above, according to the first embodiment, an image of asubject captured by the image capturing device 12 is analyzed, and theprojection image according to the analysis result of the image isemitted by the emission device 13 onto the subject. Therefore, thegraphic image appropriate for the captured image can be projected. Thefirst embodiment has been hereinabove explained.

3. Second Embodiment

Subsequently, the second embodiment will be explained. As describedabove, in the techniques in the related art, a graphic image isprojected onto a position recognized using a marker and the like inadvance. Therefore, there is an issue in that a graphic image may not beprojected onto a position directly specified by a user on the displayscreen which displays the captured image. Accordingly, an informationprocessing apparatus 20 according to the embodiment of the presentinvention has been created in view of the above circumstance as one ofissues taken into consideration. According to the information processingapparatus 20 according to the present embodiment, a desired graphicimage can be projected with intuitive operation performed by a user.

3-1. Hardware Configuration of Information Processing Apparatus

A hardware configuration of the information processing apparatus 20 willbe explained with reference to FIG. 19. FIG. 19 is a block diagramillustrating a hardware configuration of the information processingapparatus 20. As shown in FIG. 10, the information processing apparatus20 includes a display device (display) 11, an image capturing device(camera) 12, an emission device (projector) 13, a CPU 21, an inputdevice 15, a RAM (Random Access Memory) 16, a nonvolatile memory 17, atouch panel 22, a motion sensor 23, and the like. In the presentembodiment, the camera 12 and the projector 13 are arranged at fixedpositions of the casing of the information processing apparatus 20.

The same configurations as those of the first embodiment are notexplained in the explanation below. Configurations different from thefirst embodiment will be particularly explained in detail. The touchpanel 22 has two functions, i.e., display and input. The touch panel 22according to the present embodiment can detect input to the touch panel,which is given with a finger and the like of the user, and a specifiedposition is obtained. The detection method may be any method as long asposition information of a manipulating object on the display can bedetected. Examples of the detection method include a resistive filmmethod using a thin metallic constituting a transparent electrode, acapacitive method for detecting a position by finding a change ofcapacitance between a finger tip and a conductive film, an infraredlight shielding method, and an electromagnetic induction method.

In the following explanation, for example, a method for detectingoperation of the manipulating object using an electrostatic touch panelwill be explained in particular. The electrostatic touch panel haselectrostatic sensors arranged in a matrix form. The values of theelectrostatic sensors keep on changing according to changes ofcapacitances. When a finger, i.e., a manipulating object, comes intoproximity to or contact with an electrostatic sensor, the capacitancedetected by the electrostatic sensor increases. The capacitances of theelectrostatic sensors can be obtained at a time. The electrostatic touchpanel detects and interpolates changes of capacitances of all theelectrostatic sensors at a time, thereby detecting operation performedby a finger that is in proximity to or in contact with the electrostatictouch panel. The touch panel 22 outputs the values of the detectedcapacitances to the CPU 21.

The CPU 21 associates various kinds of information given by the touchpanel 22 with display positions of display contents displayed on thedisplay device 11, and analyzes movement of the manipulating object.Then, the CPU 21 recognizes input information given to the informationprocessing apparatus 10 based on analyzed movement of the manipulatingobject, and executes processing corresponding to the input information.As described above, the user can manipulate the contents displayed onthe display screen, and can input the input information.

The motion sensor 23 has a function of detecting a rotation angle and aninclination of a main body of the information processing apparatus 20.The motion sensor 23 may be, for example, an acceleration sensor and agyro sensor. The information about the rotation angle and theinclination of the information processing apparatus 20 detected by themotion sensor 23 is output to the CPU 21. The CPU 21 detects movement ofthe main body of the information processing apparatus 20 based on therotation angle and the inclination output from the motion sensor 23.

3-2. Functional Configuration of Information Processing Apparatus

The hardware configuration of the information processing apparatus 20has been hereinabove explained. Subsequently, a functional configurationof a control unit (CPU 21) of the information processing apparatus 20will be explained with reference to FIG. 20. Reference is made to FIGS.21 to 31 as necessary when the functional configuration of FIG. 20 isexplained. FIG. 20 is a block diagram illustrating the functionalconfiguration of the control unit of the information processingapparatus 20.

As shown in FIG. 20, the control unit of the information processingapparatus 20 includes an image capturing unit 202, a detection unit 204,an emission unit 206, an analyzing unit 208, a storage unit 210, anobtaining unit 212, and the like. The image capturing unit 202 has afunction of causing an image capturing device 12 to capture an image ofthe subject. Further, the image capturing unit 202 has a function ofcapturing an image of the subject onto which the projection image isemitted by the emission unit 206. The image captured by the imagecapturing unit 202 is provided to the emission unit 206 and theanalyzing unit 208.

The emission unit 206 has a function of causing the emission device 13to emit a projection image based on user's input to the subject. Theprojection image based on user's input is a projection image specifiedand selected by a user with the touch panel 22 and the like. Theprojection image may be obtained from the storage unit 108 based onuser's input, and may be emitted onto the subject.

As described above, in the present embodiment, a photo shooting regionthat can be captured by the image capturing device 12 is insynchronization with a projection region onto which the emission device13 can emit a projection image. As shown in FIG. 21, when the photoshooting region 53 and the projection region 54 are not insynchronization, the display 11 of the information processing apparatus20 may indicate the projection region. For example, shown in FIG. 21, aframe representing the projection region may be displayed on the display11, and a possible emission region may be presented to a user. In FIG.21, a possible projection region is a region 53, and a possible emissionregion is a region 54. Therefore, the frame representing the possibleemission region is displayed on the display screen.

Reference is made back to FIG. 20. The obtaining unit 212 has a functionof obtaining a specified position specified by user's input on thedisplay screen showing the subject. As described above, a user's inputis given from the touch panel 22, and the obtaining unit 212 obtains theuser's input on the display screen. The obtaining unit 212 may obtain aprojection image drawn by user's operation on the display screen showinga subject, and may select a desired projection image from amongprojection images displayed the display screen.

For example, as shown in FIG. 22, an image capturing unit 202 capturesan image of a subject 501, and a display image 502 is displayed on thedisplay screen. A user touches the display screen (touch panel 22) tospecify a desired position on the display image 502. In the presentembodiment, the touch panel 22 is used to specify a desired position onwithin the display image 502. However, the present embodiment is notlimited to such example. Alternatively, a desired position may bespecified using an input device such as a button and a lever with whicha desired position can be specified within the display image 502.

Then, the obtaining unit 212 obtains the specified position specified bya user, and as shown in an explanatory diagram 503, a projection imageis generated so that the projection image is emitted onto a specifiedposition. As shown in the explanatory diagram 503, a projection imagemay be generated so as to emit a predetermined color onto a positionspecified by a user. Then, as shown in an explanatory diagram 505, theprojection image is emitted onto a subject. As shown in an explanatorydiagram 504, the image capturing device 12 captures an image of thesubject onto which the projection image is emitted. As described above,when the storage unit 108 stores a projection image which is to beemitted, the projection image is obtained from the storage unit 108based on a user's instruction.

As shown in FIG. 23, a projection image may be emitted to exertillumination effect to a specified position specified by a user. Forexample, as shown in an explanatory diagram 516, the image capturingdevice 12 captures an image of a subject 515, and the image is displayedon the display screen. Then, when a user specifies a facial portion of asubject, a projection image is generated to exert illumination effect toa specified portion as shown in an explanatory diagram 517. Then, asshown in an explanatory diagram 519, the projection image is emittedonto the subject. As shown in an explanatory diagram 518, the imagecapturing device 12 captures an image of the subject onto which theprojection image is emitted. In an explanatory diagram 518, a projectionimage is emitted to illuminate the face of the subject.

As shown, in FIG. 24, a projection image drawn according to user'soperation performed on a display screen showing a subject may be emittedonto the subject. For example, as shown in an explanatory diagram 521,when the user writes a text “friend” on the display screen (touchpanel), the projection image is generated to emit the text “friend” tothe subject. Then, as shown in an explanatory diagram 522, theprojection image including the text “friend” written by the user isemitted onto the subject.

As shown in FIG. 25, a desired projection image may be selectedaccording to user's input from among a plurality of projection imagesdisplayed on the display screen. For example, in an explanatory diagram525, a plurality of graphic images of projection images are displayed onthe display screen. The user selects a desired graphic image on thedisplay screen (touch panel). In this case, the user may select adesired graphic image by touching the display screen, or may select adesired graphic image using an input device such as buttons and levers.

Further, as shown in an explanatory diagram 526, a position is specifiedby users input, so that the selected graphic image is emitted onto thespecified position. Then, as shown in an explanatory diagram 527, aprojection image is generated so that the selected graphic image isemitted to the specified position, and the projection image is emittedonto the subject. As described above, the user can emit the projectionimage onto the subject with intuitive operation such as directly drawingand selecting the graphic image on the display screen.

Explanation about the functional configuration of the informationprocessing apparatus 20 will be continued with reference back to FIG.20. The analyzing unit 202 has a function of analyzing an image of asubject captured by the image capturing device 202. The analyzing unit208 extracts feature points of the captured image of the subject, andprovides the information about the extracted feature points of the imageto the emission unit 206. The emission unit 206 emits a projection imageonto the subject according to the feature points of the subjectextracted by the analyzing unit 208.

Now, the extraction of the feature points of the subject will beexplained with reference to FIG. 26. For example, in an explanatorydiagram 505 of FIG. 26, feature points of an image are extracted beforepersons in the subject move, and a projection image is emitted accordingto positions of the persons. In the explanatory diagram 505, forexample, the projection image is emitted to a region close to the headsof the persons. At this occasion, as shown in an explanatory diagram507, relative positions of the feature points and the projection imageare recorded. Then, as shown in an explanatory diagram 506, featurepoints of the image are extracted after the persons in the subject move,and a projection image is emitted according to the feature points of theimage. In other words, even when the persons in the image move, theemission position is corrected based on the relative positions of thefeature points of the persons and the projection image so that theprojection image is emitted to a region close to the heads of thepersons.

Explanation about the functional configuration of the informationprocessing apparatus 20 will be continued with reference back to FIG.20. The detection unit 204 uses the motion sensor 23 to detect movementof the body of the information processing apparatus 20. As describedabove, the rotation speed and the inclination of the main body of theinformation processing apparatus 20 can be detected by the motion sensor23. Therefore, the detection unit 204 detects the movement of the mainbody of the information processing apparatus 20 based on the rotationangle and the inclination output from the motion sensor 23. Then, theemission unit 206 emits a projection image onto a subject according tothe movement of the main body of the information processing apparatus 20detected by the detection unit 204.

Now, a projection image emitted according to movement of the informationprocessing apparatus 20 will be explained with reference to FIG. 27. Anexplanatory diagram 510 of FIG. 27 is an image captured before theinformation processing apparatus 20 moves. For example, as shown in anexplanatory diagram 511, it is assumed that the information processingapparatus 20 moves to the left after the image shown in the explanatorydiagram 510 is captured. The detection unit 204 uses the motion sensor23 to detect that the main body of the information processing apparatus20 moves to the left, and provides the detection result to the emissionunit 206.

As shown in an explanatory diagram 512, the emission unit 206 correctsthe position of a graphic image emitted according to the movement of themain body of the information processing apparatus 20. In other words, aprojection image is generated in which the position of the emittedgraphic image is shifted to the left according to a distance theinformation processing apparatus 20 moves to the left. Then, as shown inan explanatory diagram 513, a graphic image is emitted to a particularposition of an image even when the information processing apparatus 20moves to either the right or left.

As shown in FIG. 28, a plurality of information processing apparatuses20 may emit different projection images onto one subject. For example,as shown in FIG. 28, an information processing apparatus 20A held by auser A and an information processing apparatus 20B held by a user B emitrespective projection images onto a subject 527. It is assumed that,first, the information processing apparatus 20A emits a projection imageas shown in an explanatory diagram 531, and then, the informationprocessing apparatus 20B emits a projection image as shown in anexplanatory diagram 532. In this case, both of the projection imageemitted by the information processing apparatus 20A and the projectionimage emitted by the information processing apparatus 20B are projectedonto the subject. Both apparatuses capture an image of the projectionimage shown in the explanatory diagram 530.

In this case, in order to successively emit different projection imagesonto one subject and capture images thereof, the photo shooting regionand the projection region need to be in synchronization (in agreement)in each apparatus. Accordingly, the image processing as shown in FIG. 26is performed to extract feature points of the subject and detectmovement of the main body of the apparatus, so that the position of theprojection region is corrected so as to bring the photo shooting regionand the projection region into synchronization with each other.Alternatively, when different apparatuses emit projection images,emission synchronization according to which a projection image isemitted from an emission device needs to be in agreement. For example,both apparatuses may be connected via a network to share informationabout emission synchronization. Alternatively, information aboutemission synchronization may be embedded in a projection image, and theinformation about emission synchronization is obtained by taking animage of the projection image.

Alternatively, different apparatuses may have emission synchronizationsin different cycles, and may capture a different projection image foreach cycle. For example, the information processing apparatus 20A andthe information processing apparatus 20B have different emissionsynchronizations. An image may be captured in the same cycle as theemission synchronization in which a projection image is emitted by theinformation processing apparatus 20A, and an image may be captured inthe same cycle as the emission synchronization in which a projectionimage is emitted by the information processing apparatus 20B, so that adifferent projection image can be captured for each cycle.

When the cycles are changed in this manner, it is possible todistinguish which apparatus emits a projection image. Alternatively, apredetermined pattern is embedded into a projection image, and theprojection image may be captured and analyzed in image processing tofind a predetermined pattern, so as to distinguish which apparatus emitsthe projection image. Alternatively, a pattern including time-seriesinformation may be embedded into a projection image, and time-seriesinformation such as a projection order of a plurality of projectionimages may be analyzed by capturing the image of the pattern.

As shown in FIG. 29, a projection image changing according to time maybe projected onto a subject. For example, as shown in an explanatorydiagram 536, a position where the projection image is emitted isspecified by a finger and the like of a user. Then, as shown in anexplanatory diagram 535, a circular projection image is emitted onto thesubject. As shown in an explanatory diagram 537, the circular projectionimage changes according to time. For example, the circular projectionimage is changed as shown in the explanatory diagram 537, and aremaining time until a shutter is released is presented to the subject.

As shown in FIG. 30, an information processing apparatus 30 may beassociated with a television receiver and a remote controller, and aprojection image specified on a touch panel of a remote controller isdisplayed on a display screen of television receiver. For example, asshown in an explanatory diagram 541, the television receiver has animage capturing device and an emission device, which respectivelycaptures an image of users who watch the television receiver and emits aprojection image onto the users. The user has a remote controller, andas shown in an explanatory diagram 540, the display screen of the remotecontroller displays the image captured by the image capturing device ofthe television receiver. Then, when the user writes a predetermined textand the like on the touch panel of the remote controller as shown in anexplanatory diagram 542, the projection image drawn by the user isemitted onto the subject as shown in an explanatory diagram 543. Then,the image capturing device of the television receiver captures an imageof the subject onto which the projection image is emitted, and the imageis displayed on the display screen of the television receiver.

As shown in FIG. 31, after an image of a subject is captured onto whicha projection image is emitted, the emitted projection image may becorrected or recomposed by image processing. For example, as shown in anexplanatory diagram 545, a projection image is emitted onto a subject,and an image of the subject is captured onto which the projection imageis emitted. Then, as shown in an explanatory diagram 546, the image ofthe subject onto which the projection image is emitted is processed toperform color correction of the projection image. Therefore, not only aphotographer but also a subject can find what kind of graphic image isoutput to which portion of the subject, and a desired composite graphicimage can be made.

3-3. Operation of Information Processing Apparatus in Detail

The functions of the control unit of the information processingapparatus 20 have been hereinabove explained. Subsequently, operation ofthe information processing apparatus 20 will be explained in detail withreference to FIGS. 32 to 34. FIGS. 32 to 34 are flowcharts eachillustrating operation of the information processing apparatus 20 indetail.

As shown in FIG. 32, first, the information processing apparatus 20determines whether a projector (emission device 13) is ON or not (S202).When the projector is determined to be ON in step S202, an imagecaptured by the image capturing device 12 is obtained (S204). The imageobtained in step S204 is displayed on the touch panel (S206). Then, adetermination is made as to whether a user specifies a position on thetouch panel showing the image (S208). In this embodiment, adetermination is made as to whether a user specifies a position on thetouch panel in step S208. However, this embodiment is not limited tosuch example. For example, whether the user specifies a position may bedetermined based on input to operation buttons and operation levers ofthe information processing apparatus 20.

When it is determined that the user specifies a position in step S208, agraphic image is generated according to the position specified by theuser (S210). When it is determined that the user does not specify aposition in step S208, a determination is made as to whether the userperforms any operation relating to a graphic image other than thespecifying position (S212). Examples of operations other than specifyingposition include selecting a desired graphic image from among aplurality of graphic images. When it is determined that the userperforms another operation relating to the graphic image in step S212, agraphic image is generated according to the operation (S214).

Then, a projection image including the graphic image generated in stepS210 or step S214 is emitted onto a subject (S216). Then, adetermination is made as to whether an image of a subject is capturedonto which the projection image is emitted (S218). When it is determinedthat photo shooting is performed in step S218, the captured image issaved (S220).

Subsequently, processing for fixing a position of an emitted projectionimage by image processing will be explained with reference to FIG. 33.In the explanation below, it is assumed that a graphic image has alreadybeen generated and a projection image is emitted onto a subjectaccording to the processing of FIG. 32. As shown in FIG. 33, first, adetermination is made as to whether the projector is ON or not (S302).When the projector is determined to be ON in step S302, an imagecaptured by the image capturing device 12 is obtained (S304). The imageobtained in step S304 is an image of the subject onto which a projectionimage is emitted.

Subsequently, feature points of the image obtained in step S304 aredetected (S306). Then, relative relationship between a projected graphicposition and the feature points detected in step S306 is recorded(S308). Then, it is collated with feature points in a previous frame(S310). Then, the drawing position of the already drawn graphic image iscorrected using the feature points collated in step S310 (S312). Theprocessings of step S306 and subsequent steps may be repeated every timean object and a person included in the subject move. Therefore, theprojection image can be emitted to a fixed position at all times withrespect to the object and the person included in the subject.

Subsequently, processing for fixing a position of an emitted projectionimage by detecting movement of the apparatus will be explained withreference to FIG. 34. In the explanation below, it is assumed that agraphic image has already been generated and a projection image isemitted onto a subject according to the processing of FIG. 32. As shownin FIG. 34, first, a determination is made as to whether the projectoris ON or not (S322). When the projector is determined to be ON in stepS322, an image captured by the image capturing device 12 is obtained(S324). Then, an output value output from a motion sensor (gyro sensor)is obtained (S326).

Then, movement of the main body of the information processing apparatus20 is estimated based on the output value obtained in step S326 (S328).Then, the position of the projection image emitted onto the subject iscorrected based on movement of the main body of the informationprocessing apparatus 20 estimated in step S308 (S320). For example, whenthe main body of the information processing apparatus 20 is determinedto move to the right and left in step S328, the emission position of theprojection image is moved to the right and left. On the other hand, whenthe main body of the information processing apparatus 20 is rotated, theemitted projection image is rotated according to rotation of the mainbody of the apparatus. Therefore, the projection image can be emitted toa fixed position with respect to the object and the person included inthe subject. The operation of the information processing apparatus 20has been hereinabove explained in detail.

As described above, according to the second embodiment, the projectionimage based on user's input is emitted onto the subject, and the imagecapturing device 12 captures the image of the subject onto which theprojection image is emitted. For example, the user operates the touchpanel 22 to specify the position onto which the projection image isemitted and select a projection image to be emitted. Therefore, withintuitive operation, a user can have a desired graphic image projectedonto a subject to be photographed. The second embodiment has beenhereinabove explained.

4. Third Embodiment

Subsequently, the third embodiment will be explained. As describedabove, the technique is disclosed in the past to use a projector toproject a graphic image onto a subject to be photographed. Further,another technique is disclosed to project a graphic image onto a deskand the like, detect a direction of fingers and shapes of hands, andcause an information processing apparatus to perform predeterminedprocessing. However, it is desired to not only perform processingaccording to a direction of fingers and shapes of hands but also detecta certain movement of a subject with respect to a projected graphicimage and execute processing in association with the detected movement.

Accordingly, the information processing apparatus 30 according to theembodiment of the present invention has been created in view of theabove circumstance as one of issues taken into consideration. Theinformation processing apparatus 30 according to the present embodimentcan execute processing according to subject's interaction with aprojection image.

An overview of the present embodiment will be explained with referenceto FIGS. 35 and 36. For example, FIGS. 35 and 36 are explanatorydiagrams each illustrating an overview of the present embodiment. Likethe first embodiment, a projection image is emitted onto a subject by anemission device 13 of the information processing apparatus 30, and animage capturing device 12 captures an image of the subject onto whichthe projection image is emitted. As shown in FIG. 35, a photo shootingregion 55 and a projection region 56 are in synchronization in thepresent embodiment. A projection image 60 including a graphic image forallowing the subject to perform predetermined operation is emitted ontothe projection region 56. Then, the image capturing device 12 capturesan image of the photo shooting region 55 onto which the projection image60 is emitted.

As shown in an explanatory diagram 62 of FIG. 36, a projection image 61including graphic images of “Yes” and “No” is emitted onto theprojection region. Then, a person included in the subject gives agesture for pointing at “Yes” in the emitted projection image. Then, theimage of the subject giving the gesture for pointing at “Yes” isrecognized, and it is determined that the subject is pointing at “Yes”.In FIG. 36, the projection image allows the subject to select whetherthe flash of the image capturing device 12 is turned on or off.Accordingly, it is detected that the subject is pointing at “Yes”, andan image capturing processing is executed with the flash turned on.

As described above, according to the in processing apparatus 30 of thepresent embodiment, the main body of the information processingapparatus 30 can be operated when the subject interacts with theprojection image emitted by the emission device (projector) 13.

4-1. Hardware Configuration of Information Processing Apparatus

The hardware configuration according to the present embodiment has thesame hardware configuration as the first embodiment shown in FIG. 3, andthe detailed explanation thereof is omitted.

4-2. Functional Configuration of Information Processing Apparatus

Subsequently, a functional configuration of the information processingapparatus 30 will be explained with reference to FIG. 37. In FIG. 37, afunctional configuration of the control unit (CPU 34) of the informationprocessing apparatus 30 will be explained in particular. Reference ismade to FIGS. 38 to 49 as necessary when the functional configuration ofFIG. 37 is explained. As shown in FIG. 37, the control unit of theinformation processing apparatus 30 includes a recognizing unit 302, anemission unit 304, an execution unit 306, a storage unit 308, and thelike. The recognizing unit 302 has a function of recognizing operationof the subject to be photographed.

The recognizing unit 302 recognizes operation of the subject based onthe image of the subject captured by the image capturing device 12. Inthe explanation below, the image of the subject captured by the imagecapturing device 12 is analyzed, and operation performed with a gesturegiven by the subject is recognized. However, the present embodiment isnot limited to such example. For example, an infrared light may beemitted onto a subject, and at gesture given by the subject may berecognized using reflection result of the infrared light.

The emission unit 304 has a function of emitting a projection image ontoa subject so as to allow the subject, to be photographed, to perform acertain operation. The emission unit 304 may also emit a projectionimage so as to allow the subject to perform a certain selectionoperation. The emission unit 304 may also emit a projection imageaccording to subject's operation recognized by the recognizing unit 302.The emission unit 304 obtains a projection image from the storage unit308, and emits the projection image onto the subject. The emission unit304 may obtain a projection image from the storage unit 308 according touser operation, and may obtain a projection image according to a personand an object included in a subject from the storage unit 308.

When the recognizing unit 302 recognizes a person and an object includedin a subject, the emission unit 304 may emit a projection image onto aregion of the person and the object included in the subject, or may emita projection image onto a region other than the person and the object.Further, when the recognizing unit 302 recognizes a face of a particularperson included in a subject, a projection image may be emitted to aregion around the particular person.

For example, as shown in an explanatory diagram 601 of FIG. 38, when thefaces of persons included in a subject are detected, a projection imagemay be emitted onto a region other than the regions of the faces of thepersons. Further, as shown in an explanatory diagram 602, when the faceof an owner of the information processing apparatus 30 can be detectedfrom among the faces of the persons, a projection image may be emittedto a region around the face of the owner. Accordingly, the personsincluded in the subject can easily perform operation on the projectionimage.

Reference is made back to FIG. 37. The execution unit 306 has a functionof executing predetermined processing according to subject's operationrecognized by the recognizing unit 302. The execution unit 306 executesprocessing according to a gesture given by a subject, which isrecognized as the recognizing unit 302. For example, when a gesture forturning on flash is recognized as described above, the flash of theinformation processing apparatus 30 is turned on, and an image capturingprocessing is executed. When the recognizing unit 302 recognizesmovement of subject in a range onto which a projection image is emittedby the emission unit 304, processing corresponding to the movement ofthe subject in the emitted range may be executed.

For example, as shown in FIG. 39, a projection image shown in anexplanatory diagram 606 is emitted onto the subject. In this case, asshown in an explanatory diagram 605, the projection image shown in theexplanatory diagram 606 is emitted onto the subject. Then, as shown inan explanatory diagram 607, the image capturing device 12 captures animage of the subject onto which the projection image is emitted. At thisoccasion, the recognizing unit 302 recognizes only the region onto whichthe projection image is emitted. Therefore, the gesture given by thesubject can be reliably recognized. That is, the recognizing unit 302does not need to recognize predetermined operation in all of capturedimages. The recognizing unit 302 only needs to recognize predeterminedoperation included in a region 608. Therefore, processing load can bereduced.

Subsequently, an example of operation (interaction) performed by asubject on an emitted projection image be explained with reference toFIGS. 39 to 48. For example, as shown in FIG. 40, when the recognizingunit 302 detects a certain movement, the recognizing unit 302 mayrecognize that operation is performed on a projection image. Forexample, in an explanatory diagram 610, an icon of a graphic image of acamera is emitted, and when movement of a hand is detected above theprojection image of the icon, it is recognized that operation isperformed on the projection image. When the movement of the hand abovethe projection image of the icon is detected, the color of theprojection image of the icon is made deeper or the color is changed asshown in an explanatory diagram 611.

As shown in FIG. 41, when a hand coming into the projection region towhich the icon is projected is detected, the recognizing unit 302 mayrecognize that operation is performed on the projection image. Whetherthe hand enters the projection region can be determined based on, forexample, whether an object substantially having flesh color comes intothe region or not. For example, in an explanatory diagram 615, an iconof a graphic image of a camera is emitted onto a subject, and adetermination is made as to whether a hand comes into a projectionregion of the projection image of the icon and stays there for a certainperiod or more. For example, as shown in an explanatory diagram 615,after the recognizing unit 302 recognizes that the hand comes into theprojection region of the projection image of the icon, the color of theprojection image is made deeper or the color is changed in a stepwisemanner as the time passes.

FIG. 42 shows a case where a projection image is emitted to allow asubject to make a choice from two options. For example, as shown in anexplanatory diagram 620, when a hand coming into the projection regionto which the icon is projected is detected, a projection image forallowing a subject to make a choice from two options is emitted onto thesubject. For example, as shown in an explanatory diagram 621, theprojection image including “Yes” on the left side and “No” on the rightside of the icon is emitted. Then, as shown in an explanatory diagram622, when the hand is determined to move to the right side from theregion onto which the icon is projected, it can be determined that “Yes”is selected. The execution unit 306 executes processing according to theselection. For example, when whether the flash is turned on or not is tobe selected, an image capturing processing is executed with the flashturned on.

FIG. 43 shows a case where a projection image is emitted to allow asubject to make a choice from a plurality of options. For example, asshown in an explanatory diagram 625, when a hand coming into theprojection region to which a circular icon is projected is detected, aprojection image for allowing a subject to make a choice from aplurality of options is emitted onto the subject. For example, as shownin at explanatory diagram 626, the projection image including optionsshown in a radiating manner from the circular icon is entitled. Then, asshown in an explanatory diagram 627, when a hand is determined to moveto any one of the regions of the options from the region onto which theicon is projected, the option in the region where the hand is placed canbe recognized as the selected option. The execution unit 306 executesprocessing according to the selection. For example, when a shooting modesuch as white balance and shutter speed for photo shooting is selectedin a stepwise manner, an image capturing processing is executed in theselected shooting mode.

FIG. 44 is a case where a projection image is changed according tomovement of a subject. For example, a shown in an explanatory diagram630, it is assumed that a hand coming into the projection region towhich a rectangular icon is projected is detected, and thereafter, aclenched fist as shown in an explanatory diagram 631 is detected. Whenthe recognizing unit 302 detects movement from an open hand to aclenched fist, the recognizing unit 302 determines that “grasping”action is performed. Then, as shown in an explanatory diagram 632, whena hand is moved while the “grasping” action is performed, the projectionit tracks the movement. Then, as shown in an explanatory diagram 634,when it detected that the state of the hand is changed from a clenchedstate to an opened slate, the tracking of the projection image isterminated.

FIG. 45 shows a case where a hand of a subject is detected, and aprojection image is emitted onto the detected hand. For example, asshown in an explanatory diagram 640, the palm of the subject isdetected. Then, as shown in an explanatory diagram 641, the circularprojection image is emitted onto the region of the palm of the detectedhand. In this case, the recognizing unit 302 keeps on detecting theregion of the palm of the hand, and the emission unit 304 emits theprojection image onto the region of the palm so as to track the movementof the palm of the hand.

Then, as shown in an explanatory diagram 642, when the palm of the handmoves quickly to go out of the region onto which the circular projectionimage is emitted and go far from the region, this may be recognized asoperation performed on the projection image for canceling a certainprocessing. As shown in an explanatory diagram 643, when movement forclenching the palm of the hand is detected, this may be recognized asoperation performed on the projection image for deciding a certainprocessing.

FIG. 46 shows a case where palms of two hands are detected, and aprojection image is emitted onto the detected two hands. For example, asshown in an explanatory diagram 645, the palms of the two hands of thesubject are detected. Then, as shown in an explanatory diagram 646, aprojection image is emitted so as to show different options on the palmsof the two hands, respectively. Then, as shown in an explanatory diagram647, when movement for clenching any one of the hands is detected, thisis recognized as operation for selecting any one of the options.

FIG. 47 shows a case where a projection image is changed according torough movement of a subject. For example, as shown in an explanatorydiagram 650, the positions of both hands and the direction of the bodyof the subject are detected, so that movement of the subject is roughlydetected. In the explanatory diagram 650, it is detected that the leftarm is stretched, and the right arm is bent. For example, whenprocessing is executed to change illumination light according tomovement of the subject, a filter for representing the illuminationlight may be moved according to the position of the detected hand asshown in an explanatory diagram 651.

FIG. 48 shows a case where a shooting field angle is changed accordingto opposite-like movement of a subject. For example, as shown to anexplanatory diagram 655, the positions of both hands and movement of thehands of the subject are detected, and a projection image representingthe shooting field angle is emitted. As described above, movement of thepalms of the hands is detected so as to detect operation for “grabbing”the emitted shooting field angle and operation for “releasing” theshooting field angle. Then, as shown in an explanatory diagram 656, whenthe subject grabs and moves the shooting field angle of the projectionimage, the shooting range is moved.

Accordingly, zoom-in and zoom-out can be performed during photo shootingaccording to operation performed by the subject. Zoom-in and zoom-outprocessing may be performed in response to not only the rough operationof the subject as shown in FIG. 48 but also reciprocal movement of ahand. An image obtained as a result of zoom-in or zoom-out may beprojected onto a region different from the region where the subject islocated.

FIG. 49 shows a case where a projection image is emitted to allow asubject to perform a predetermined operation according to operation of aphotographer. For example, in a case where the information processingapparatus 30 is arranged with a touch panel like the second embodiment,a projection image may be emitted according to a photographer's input tothe touch panel. As shown in an explanatory diagram 670 of FIG. 49, aphotographer touches the touch panel of the information processingapparatus 30 to specify a position to which a projection image isemitted. Then, as shown in an explanatory diagram 671, the projectionimage is emitted to the position specified by the photographer.

4-3. Operation of Information Processing Apparatus in Detail

The functions of the control unit of the information processingapparatus 30 have been hereinabove explained. Subsequently, operation ofthe information processing apparatus 30 will be explained in detail withreference to FIG. 50. FIG. 50 is a flowchart illustrating operation ofthe information processing apparatus 30 in detail.

As shown in FIG. 50, first, a determination is made as to whether theprojector (emission device 13) is ON or not (S402). When the projectoris determined to be ON in step S402, a determination is made as towhether the projector emits a GUI or not (S404). The GUI emitted in stepS404 is a projection image for allowing a subject, of which image is tobe captured, to perform a predetermined operation.

When the projector is determined to emit the GUI in step S404, an imagecaptured by the camera (image capturing device 12) is obtained (S406).Then, a gesture recognition is performed only in a region showing theGUI emitted in step S404 (S408). As described above, when the projectoremits the GUI of the projection image for allowing the subject toperform a predetermined operation, a certain operation is performed onthe emitted GUI. This means that it is sufficient to recognize operationof the subject only in the region onto which the GUI is emitted.

When the projector n determined not to emit the GUI in step S404, theimage captured by the camera (image capturing device 12) is obtained(S410). Then, a gesture recognition is performed (S412). In step S412,operation performed by the subject on all over the image obtained instep S410 is recognized.

Then, whether a gesture is recognized or not is determined in step S408or step S412 (S414). When it is determined that the gesture recognitionis performed in step S414, operation is executed according to thecontent of the recognized gesture and the position where the gesture isgiven (S416). For example, the content a the gesture may be closing thepalm of an open hand and pointing the fingers of a hand to a certaindirection. For example, when a projection image including texts of “Yesand “No” is emitted, and the hand of the subject moves to a region of“Yes”, operation is executed according to “Yes”. For example, the flashis turned on.

Then, the GUI is changed or a new GUI is projected so as to reflect theresult of operation executed in step S416 (S418). Then, a determinationis made as to whether photo shooting is performed or not (S420). When itis determined that photo shooting is performed in step S420, theprojection image (GUI) which is to be operated by the subject is erasedbefore photo shooting is performed (S422). In step S422, erasing the GUImeans that the emission device 13 no longer emits the projection image.Then, photo shooting is executed after the GUI is erased in step S422(S424). The operation of the information processing apparatus 30 hasbeen hereinabove explained in detail.

As described above, according to the third embodiment, the projectionimage for allowing the subject, which is to be photographed, to performa predetermined operation is emitted onto the subject, and operationperformed by the subject on the emitted projection image is recognized.Then, the predetermined processing is executed according to therecognized operation performed by the subject. Therefore, operationperformed on the formation processing apparatus 30, such as photoshooting processing, can be executed according to subject's interactionwith the projection image.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, each step in the processings performed by the informationprocessing apparatuses 10, 20, 30 in this specification may not benecessarily processed in time-series according to an order described inthe flowcharts. For example, the steps of the processings performed bythe information processing apparatuses 10, 20, 30 may be performed in adifferent order from the order described in the flowcharts, or may beperformed in parallel.

A computer program for causing hardware such as a CPU, a ROM, and a RAMin the information processing apparatuses 10, 20, 30 to achieve the samefunctions as the constituent elements of the information processingapparatuses 10, 20, 30 may be generated. In addition, a storage mediumstoring the computer program is also provided.

What is claimed is:
 1. An information processing apparatus comprising:circuitry configured to detect a position of a subject in a capturedimage, recognize an operation performed by the subject in the capturedimage, cause an additional image to be added onto the subject in thecaptured image in accordance with the detected position of the subject,and generate a composite image in which the additional image is addedonto the subject in the captured image.
 2. The information processingapparatus according to claim 1, wherein the circuitry is furtherconfigured to detect a particular region of the subject in the capturedimage, and cause the additional image to be added onto the particularregion of the subject in the captured image.
 3. The informationprocessing apparatus according to claim 1, wherein the circuitry isfurther configured to detect a particular region of the subject in thecaptured image, and cause the additional image to be added onto thesubject in the captured image to avoid the particular region.
 4. Theinformation processing apparatus according to claim 1, wherein thecircuitry is further configured to analyze the captured image, and causethe additional image, having a color selected based on a result ofanalyzing the captured image, to be added onto the subject in thecaptured image.
 5. The information processing apparatus according toclaim 1, wherein the circuitry is further configured to analyze thecaptured image, and cause the additional image to be added onto thesubject in the captured image in accordance with a brightness of thecaptured image.
 6. The information processing apparatus according toclaim 1, wherein the circuitry is further configured to analyze thecaptured image and at least one additional image that is temporallyadjacent to the captured image to detect a movement of the subject, andcause the additional image to be added onto the subject in the capturedimage in accordance with the detected movement of the subject.
 7. Theinformation processing apparatus according to claim 1, wherein theadditional image comprises a shading pattern.
 8. The informationprocessing apparatus according to claim 1, wherein the operation is agesture made by the subject in the captured image.
 9. An informationprocessing method being performed by an information processing apparatushaving a processor, the method comprising: detecting a position of asubject in a captured image; recognizing an operation performed by thesubject in the captured image; causing an additional image to be addedonto the subject in the captured image in accordance with the detectedposition of the subject; and generating a composite image in which theadditional image is added onto the subject in the captured image. 10.The information processing method according to claim 9, furthercomprising: detecting a particular region of the subject in the capturedimage; and causing the additional image to be added onto the particularregion of the subject in the captured image.
 11. The informationprocessing method according to claim 9, further comprising: detecting aparticular region of the subject in the captured image; and causing theadditional image to be added onto the subject in the captured image toavoid the particular region.
 12. The information processing methodaccording to claim 9, further comprising: analyzing the captured image;and causing the additional image, having a color selected based on aresult of analyzing the captured image, to be added onto the subject inthe captured image.
 13. The information processing method according toclaim 9, further comprising: analyzing the captured image; and causingthe additional image to be added onto the subject in the captured imagein accordance with a brightness of the captured image.
 14. Theinformation processing method according to claim 9, further comprising:analyzing the captured image and at least one additional image that istemporally adjacent to the captured image to detect a movement of thesubject; and causing the additional image to be added onto the subjectin the captured image in accordance with the detected movement of thesubject.
 15. The information processing method according to claim 9,wherein the additional image comprises a shading pattern.
 16. Theinformation processing method according to claim 9, wherein theoperation is a gesture made by the subject in the captured image.
 17. Anon-transitory computer-readable storage medium having stored thereon aprogram, which when executed by a computer causes the computer toexecute a method, the method comprising: detecting a position of asubject in a captured image; recognizing an operation performed by thesubject in the captured image; causing an additional image to be addedonto the subject in the captured image in accordance with the detectedposition of the subject; and generating a composite image in which theadditional image is added onto the subject in the captured image. 18.The non-transitory computer-readable storage medium according to claim17, wherein the executed method further comprises: detecting aparticular region of the subject in the captured image; and causing theadditional image to be added onto the particular region of the subjectin the captured image.
 19. The non-transitory computer-readable storagemedium according to claim 17, wherein the executed method furthercomprises: detecting a particular region of the subject in the capturedimage; and causing the additional image to be added onto the subject inthe captured image to avoid the particular region.
 20. Thenon-transitory computer-readable storage medium according to claim 17,wherein the executed method further comprises: analyzing the capturedimage; and causing the additional image, having a color selected basedon a result of analyzing the captured image, to be added onto thesubject in the captured image.
 21. The non-transitory computer-readablestorage medium according to claim 17, wherein the executed methodfurther comprises: analyzing the captured image; and causing theadditional image to be added onto the subject in the captured image inaccordance with a brightness of the captured image.
 22. Thenon-transitory computer-readable storage medium according to claim 17,wherein the executed method further comprises: analyzing the capturedimage and at least one additional image that is temporally adjacent tothe captured image to detect a movement of the subject; and causing theadditional image to be added onto the subject in the captured image inaccordance with the detected movement of the subject.
 23. Thenon-transitory computer-readable storage medium according to claim 17,wherein the additional image comprises a shading pattern.
 24. Thenon-transitory computer-readable storage medium according to claim 17,wherein the operation is a gesture made by the subject in the capturedimage.